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ABSTRACT: Safety management in construction sites has always been one of the most sensitive aspects of the 
AECO industry and a problematic that recalls the complexity of such a multifactor domain. The high number of 
work accidents that occur on construction sites is also caused by the fact that not all the information to work safely 
is always available. For instance, visibility during some maneuvers is a key aspect of safety in operations, and this 
is often impeded due to the layout of the construction site and working methods, especially in the use of some 
equipment. The latest approaches in order to overcome complex situations is represented by the Digital Twin 
paradigm. This approach has among its main criticisms: 1) the way of connecting physical reality and its digital 
replica and 2) the system for exploiting the combination of real-time data and digital applied intelligence for 
supporting operations on site. This paper proposes a framework for the development of digital twin of the 
construction site. An application of augmented reality that exploits the concept of diminished reality and workers 
location detection will improve visibility during critical operations. 
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1. INTRODUCTION 


Safety in construction has always been a major issue that impacts the industry in terms of efficiency and cost. 
Construction is in fact one of the most hazardous industries due to its dynamic, temporary, and decentralized nature 
(Li et al., 2015). Among all economic activities, with the exception of oil, coal, and mineral extraction, the highest 
shares of serious incidents are recorded precisely in the construction divisions, which record rates above 35 
percent. In the five-year period 2014 - 2018, compared to cases that occurred in other sectors, events in construction 
showed a higher prevalence of risk factors related to the predisposition of work environments (19% vs 12%) and 
procedures implemented by the injured (40% vs 47%) (Inail, 2022). These statistics focus our attention on two 
aspects: 1. The worksite is a place that by its conformation and nature carries a high percentage of risk; 2. The 
proper execution of procedures is a critical aspect that should be worked on to reduce its percentage of risk that is 
the highest among all (Li et al., 2015). 


The traditional approach to safety also referred to as Safety-I (Martins et al., 2022) involves working to reduce the 
number of adverse events as much as possible. This most often involves using methods to counter or prevent the 
occurrence of misbehavior (e.g., audible warning systems on machinery) or to limit the effects should they 
eventually occur (e.g., individual and collective prevention devices). This approach involves having to envision 
all possible adverse scenarios so as to provide methods to counteract possible adverse outcomes. This is a 
continuous chase that however struggles to take into account a fundamental aspect that characterizes construction 
sites: their high complexity (Fang and Wu, 2013). Difficulty in standardization of procedures, complicated 
production processes, temporary organizational structure, relationships between stakeholders, and variable 
workplaces are some of the factors that contribute to the deep diversification between construction sites and other 
industrial production sites (Li et al., 2015). All these factors provide the backdrop for another crucial aspect that 
is encompassed in the concept of complexity and that is the need to manage the unexpected, which in terms of 
safety can be translated as dealing with new, variable and real-time risks and hazards. 


In this context, the use of new technologies to collect and analyze real-time data from the construction site can be 
a disruptive innovation (Teizer et al., 2022). A push in this direction is also provided in the Italian context by Law 
36/2023, which in Annex 1.9, article 12 letter m states: “Jn the formulation of information requirements by 
contracting stations, specific uses, operational methodologies, organizational processes and technological 
solutions may be defined as objects of evaluation for the purpose of rewarding, with reference to the execution 
phase of works, to digitally increase health and safety conditions at construction sites. ” However, there are still 
unresolved issues in this regard that prevent widespread and pervasive use of these methods in the real 
environment. On the one hand, real-time on-site data collection itself encapsulates within it several problems: the 
sensorization of construction sites (workers, vehicles, materials), methods for collecting images that can be 
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exploited to apply artificial intelligence, the possible need to have to provide power in the open environment to 
IoT devices, and the lack of a unified framework where to channel and integrate the collected data. On the other 
hand, information derived from analyses, short-term simulations or application of artificial intelligence should 
increase the knowledge of the stakeholders involved and thus in the interest of safety management they should be 
made immediately available on site in order for being used by operators, safety managers and site managers. This 
aspect represents a great complexity because making content usable on site requires finding quick and effective 
methods of communication both in terms of sending the data and visualizing it at the construction site. Augmented 
reality is a technology that has great potential in terms of visualization. Indeed, the use of holograms superimposed 
on the real-world view makes it possible to add an information layer that can show aspects not visible to the human 
eye or enrich knowledge with information from analysis of previously submitted data. In some cases, however, 
what is needed is not so much to add a layer of knowledge as to remove something that impedes the view. There 
is for this purpose a specific class of augmented reality that is called diminished reality (Cheng at al., 2022). 


The purpose of this research is to apply the concept of diminished reality to construction sites in order to support 
operations with regard to risks related to operator safety. The proposed framework and the first experiments 
developed involved moving loads with a crane work scenario, including work at height, where the operator of the 
vehicle has a not fully unobstructed view. The application of see-through diminished reality allows the crane 
operator to visualize any worker placed in the vicinity of the load through the use of a depth camera and artificial 
intelligence. Innovations of the proposed system include: 1. Visualization by the operator not only of the relative 
position between workers and load but also of the orientation of the workers' bodies. This information makes it 
possible to understand whether the worker is aware that he or she is near a moving load or whether there is a risk 
of catching the worker by surprise (in case it is shown that the subject is with his or her back to the load). 2. An 
automatic procedure that transforms the visualization from solid to wireframe of objects that are opposed between 
the machine operator and human figures detected by the depth camera. In this way through a two-way line of 
communication and interaction between the actual construction site and a digital representation of it, a Digital 
Twin (DT) type approach will be achieved in which data obtained are processed and sent back, after the necessary 
analysis, directly to the site real time use. The system proposed in this article makes more sense at large, complex 
construction sites where the expected safety costs are higher and therefore allow for the implementation of new 
technologies. 


2. LITERATURE REVIEW 


The use of innovative technologies such as virtual (VR), augmented (AR) and mixed reality (MR) in the 
construction industry has found different levels of application due to the differences between them. Applications 
using augmented and mixed reality have long since found their use in construction sites. Chalhoub et al., 2018 
proposed a method for electrical system component's location visualization on site using mixed reality and head- 
mounted displays. A similar way of exploiting MR is proposed in recent times also by Dallasega et al. (2023) for 
MEP components installations in general. A number of applications of AR and MR in construction sites still refers 
to way for showing design information or building components specifications in a more straightforward and easy 
understanding way (Carbonari et al., 2022; Yoon et al, 2022; Sabzevar et al, 2023; Pendersen et al, 2020; Um et 
al., 2023). 


As far as worksite safety is concerned, however, most applications focus on the possibilities of leveraging these 
technologies for operator training. Boschè et al. (2015) proposed a novel MR system uniquely targeted for the 
training of construction trade workers. One of the aims of this paper was to enable trainees to experience 
construction site conditions, particularly being at height, in different settings. Anyway, the majority of these 
applications makes use of VR or Augmented virtuality (Wolf et al, 2022) which means that the operator is 
transported completely into the virtual world to carry out serious game-like experiences but without testing in the 
real world or with operations carried out live. Jelonek et al. (2022) developed a VR application for training 
operators in the use of cutting equipment on site introducing in the serious game procedures to follow in order to 
be sure that all the involved personnel is following all safety prescriptions. Wolf et al. (2022) developed a serious 
game for inspector job simulation in a complete virtual environment. Speiser and Teizer (2023) tried to move 
forward introducing the concept of digital twin for construction safety training in order to link various data sources 
to generate a Virtual Training Environment (VTE) automatically. However, few applications are yet attempting to 
bring these technologies on site to provide real-time assistance. Nguyen et al. (2022) started to work on skeleton 
recognition for action recognition on site but they have not developed an AR visualization on site yet. Eiris Pereira 
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et al. (2019) combined training and augmented reality but directly on site for reporting situations with danger of 
falling from height. 


One aspect that is still underemphasized in the construction world is that sometimes it can be very useful that 
information is taken away rather than added, and in this sense the application that should be considered is that of 
Diminished Reality (DR) (Mann et al., 2002). While AR and MR superimpose virtual objects on the real world to 
enhance reality by placing new objects among real objects or extending real objects with virtual objects (Mori et 
al.,2017), DR deliberately removes parts of a real-world scene or replaces them with computer generated 
information (Mann & Fung, 2001) and it can be considered a subtype of AR. Based on computer vision techniques, 
unwanted image elements are detected and replaced by other image elements, creating an overall plausible and 
consistent impression for the viewer. The idea in diminished reality is to virtually remove something from the 
view. There can be identified different techniques for diminished reality implementation. In one kind of approach 
the object to be removed needs to be detected and the corresponding image area needs to be filled in with a texture 
that seems to belong to the background. In image processing terms, this kind of filling-in operation is called image 
inpainting (Siltanen, 2017). Another approach discusses a way of representing occluding objects as semi- 
transparent. This visualization technique is called AR X-ray vision, see-through vision, or ghosted views. Semi- 
transparent representation is useful for seeing through car interiors and walls (Mori et al., 2017). In automotive 
settings, a number of see-through displays have been incorporated. In Samsung’s Safety Truck, for example, live 
video images were displayed on the back of the truck, effectively granting trailing drivers the ability to ‘see 
through’ the truck. 


The very first approaches to DR aimed to lower the saturation of some areas to force an observer to face other 
regions and virtual objects overwrote undesirable real objects to hide the real information (Mann, 1994). After that 
this technique was used for different purposes: to remove a person from Google Street View pictures to protect his 
or her privacy, to remove a person in a video, to remove a vehicle in front of the driver, to remove a baseball 
catcher to visualize the view of the pitcher from a view behind the catcher, and to generate a panoramic 
stroboscopic image (Mori et al, 2017). In the case of see through applications one of the methods to provide the 
current state of the hidden area in the main view is the same as that used by Zokai et al. (2003) that used two 
additional cameras as hidden background observers to erase from the main view pipes in a factory. Mori et al. 
(2017) constructed light fields with a real time multi-camera system and removed a viewer’s hand from the 
perspective to visualize the viewer’s workspace occluded by his or her own hand. Queguiner et al. (2018) presented 
a diminished reality application running live on consumer mobile devices. In their pre-observation-based approach, 
the clean 3D scene, free of undesired objects, is scanned beforehand and reconstructed as a high-resolution textured 
3D model. Many see-through DR literature tends to investigate more computationally efficient approaches 
focusing on compensating real—virtual boundary in screen space. Thereafter, semi-transparent or wireframe 
representation is performed to improve user depth perception. These representation methods will be useful for 
avoiding the danger of a collision with the diminished objects. In this regard, Peereboom et al., 2023 presented a 
system that exploits DR for avoiding collisions between pedestrians and cars caused by poor visibility, such as 
occlusion by a parked vehicle. In one of the solutions, they proposed the occluding vehicle has been made semi- 
transparent. Still recently, Cheng et al. (2022) proposed a study on users' perception of diminished reality and its 
possible applications. 


However, the concept of DR is still little exploited in construction although Klinker, Stricker and Reiners (2001) 
develop the first examples of diminished reality in the field of construction with two very significant examples: in 
the first one TV antennas located on a hill are removed from the view within which the work that will replace them 
is later placed so that it can be visualized in its chosen location. In the second case given as an example, the 
installations below a wall are shown in the form of holograms. 


This research aims at exploiting the concept of diminished reality for construction sites operations efficiency and 
safety. The innovation of the system proposed lays in the combination of skeleton recognition, which provide 
operators position and field of view, and the automatic procedure that detect obstacles and made them visualized 
as a wire frame model. The integration of different technologies for onsite application is also another key aspect. 


3. DR FOR CONSTRUCTION SITE SAFETY MANAGEMENT SUPPORT 


The system proposed in this research is a diminished reality application for supporting safety management at 
construction sites. The development focused on a specific case and that is load handling by crane. In the case of 
cranes maneuvered from the ground, but not only, there are situations in which visibility can be reduced: work at 
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height in which the support is not clearly visible, maneuvers on the construction site in which the placement of 
material takes place beyond already built constructions. For this reason, a viewer that supports the operations 
coordinator in visualizing possible hazardous situations can assume great value (Fig. 1). To this end, we focused 
in this work on visualizing workers in locations hidden from view. 


It was decided to proceed with localization and visualization through depth camera with a twofold advantage: on 
the one hand, this instrument returns the skeleton that allows us to understand how the worker is placed and 
oriented; on the other hand, the fact that only the skeleton is communicated removes a number of privacy issues. 
In this system then it was planned to also instrument moving loads, with sensors (UWB inside and RTK outside) 
in order to be able to communicate this data to the operations coordinator as well. Finally, the developed application 
involves the transformation of the obstacle object visualization from solid to wireframe. Such a system finds its 
application not only in the highlighted case but also in many other site operations such as demolitions where 
knowing what lies beyond certain elements could be crucial. 


Fig. 1: Diminished reality for construction site. 


4. METHODOLOGY 


The methodology followed in this research for improving the safety of workers in construction sites through the 
exploitation of DR is shortly described in Figure 2. 


The first step is setting up a localization framework covering the entire construction site. Such framework should 
be a real-time system capable of attaching spatial coordinates to any entity, object or person, moving around in the 
construction site. The specific technology employed for this step can vary depending on several aspects, e.g. 
whether the construction site is outdoor or indoor, or whether we are considering an existing facility to be 
recommissioned or instead a new building or infrastructure to be built. For complex scenarios, multiple 
technologies may coexist at the same time to localize objects and people in different areas of the overall 
construction site. In such a case a problem arises of allowing a seamless integration of different localization 
systems to provide a homogenous notion of localization on the construction site. The core idea behind the 
localization framework is to support the detection of safety hazards so as to prevent incidents. 


Asecond step of the methodology is the setup of a heterogeneous network of sensors continuously monitoring the 
activities in the construction site. The sensor network may employ very different devices both for their 
technological characteristics and for the kind of data collected (e.g. temperature sensors, cameras, NFC tags, etc.). 
The basic requirement for each sensor in our methodology is that it must be able to attach a precise timestamp and 
a precise set of coordinates relative to the localization framework set up. This would witness when and where the 
information was sensed in the construction site. 
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A third step is the setup of a DT Platform integrating several sources of information operating on and off the 
construction site. For instance, interpolating information coming from the construction design together with data 
collected by the network of sensors the platform can provide the most updated “picture” of the building or 
infrastructure being realized. The DT Platform is also responsible for managing and harmonizing the several data 
models used to express the information coming from the different (internal and external) sources, e.g. by using 
linked data or ontology-based semantic data integration procedures. 


Monitor objects Integrate Obstacle 
and workers information recognition Rendering / 


Set up 
localization 


framework using sensors using BIM using Spatial Visualization 


network platform Engine 


Fig. 2: Methodology for applying Diminished Reality to construction site safety. 


Next, a Spatial Engine should be implemented that processes all the spatial information from the BIM models, the 
persons equipped with DR viewers, the sensors deployed through the site, the vehicles and all the various agents 
acting within the site. Given the position of a person using a DR viewer, its main role is to determine which agents 
and BIM objects are of interest for him/her and to provide the corresponding data streaming. Moreover, by 
exploiting the information stored in the CDE, the Spatial Engine is the place where possible spatial interferences 
can be detected, forecasted, and notified at run time to the involved agents in order to prevent injuries. 


Finally, a DR visualization tool is in charge of rendering holograms mixed with the real objects of the construction 
site in order to allow the observer to view also those hidden objects that can be involved by the ongoing operations. 
The obstruction is determined in real-time based on the observer’s point of view and the position of the moving 
objects given by the sensory data: when a BIM object is hit by the ray joining the observer with a moving object, 
this BIM object is classified as obstructing and it is rendered as wireframes thus making visible the virtual objects 
behind it that reproduce their invisible real twins. 


In the following sections this methodology is applied starting with the indoor localization framework. The sensor 
network has been set up with a depth camera and two location tags. The former is a special computer camera with 
two “eyes”. By merging the visual information coming from them, it can compute the distance (or depth) of objects 
and bodies w.r.t. its point of view, when they appear in the picture. The chosen depth camera also provided 
advanced artificial vision software capable of recognizing the position of core joints in the skeletons. This 
information can be exploited to know the real-time position and orientation of an individual or of a crew of 
workers. As a consequence, it is possible to relate this information the one coming from the tag attached to the 
load lifted by the crane in order to know the exact relative position and the risk of the workers to be hit. A mixed- 
reality head-mounted display (HMD) is used as visualization tool where the BIM model of the construction site 
and signals coming from the sensor network (e.g. the position of the workers in the construction site coming from 
the depth camera, as well as the position of the depth camera itself, and of the crane load) are downloaded from 
the DT platform together with the spatial simulation results. Also the head-mounted display DR application can 
download from the DT platform information about the 3D objects that hinders the view of workers and the load 
when the crane is operating thank to ray-tracing data transmission. The ray-tracing technique uses basic geometry 
in order to select all the 3D objects (e.g. walls, doors, columns, etc.) that lay in between the user viewpoint and the 
objects returned as obstructions (i.e. the workers and the crane load) and then the platform sent them back made 
transparent in a wire frame visualization. 


5. SYSTEM ARCHITECTURE 


The architecture of the system proposed in this paper is the one depicted in Figure 3. In accordance with what was 
previously described in the methodology paragraph in this section the specific devices and components of the 
system will be made explicit. For what concerns the localization of the operators, we chose to use a depth camera 
and specifically a ZED 2i. This device made possible two things: 1. recognize the skeleton of the workers; 2. place 
the skeleton in the virtual space once the position of the camera in the real space is known. The ZED 2i camera 
provides the streaming of the joints of the skeleton, which were set equal to 18 points. 
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Hololens 2 - MR headset 
Fig. 3: System architecture and data flow. 


For space modelling it can be used a BIM model provided in ifc format. This type of space modelling contributes 
in modelling the different building components enabling at a later stage to go and select one by one the objects 
that represent obstacles to workers visualization. In the development here proposed, Autodesk Revit was used for 
BIM modelling. Moreover, in the case of the construction site where the presence of the elements is dependent on 
the progress of the work, the combination of ifc files with a timeline also makes it possible to analyse what would 
be visible and what would not be visible at a precise moment in the construction. 


Finally, since not only the workers need to be located but there are other components of the worksite whose location 
is important to know, a number of sensor networks can be implemented. These vary as needed. In the case taken 
as an example in this article, two types of sensors were referred to: GPS RTK tags and UWB tags. Although the 
GPS RTK localization system could not be integrated into the indoor tests reported in section 6, its suitability for 
outdoor applications is shown by its quite high accuracy reported by technical sheets [Datasheet], which was 
confirmed by very preliminary trials performed by the authors. But this is going to be used in future research 
developments. For the purpose of the indoor tests subject of this paper, contextual infrastructure was monitored by 
means of UWB anchors. In any case, it is worth remarking that both types of sensors can be used to communicate 
position data. 


Moving on to the other components of the system, a platform, resulting from the development of the research 
project "A Distributed Digital Collaboration Framework for Small and Medium-Sized Engineering and 
Construction Enterprises" (PRIN 2017), is used. This platform uploads BIM models in ifc format, allowing 
browsing and querying. The platform then allows the integration of data from a variety of heterogeneous sources 
(in this case skeleton joints from ZED 2i and location data from sensors) allowing them to be located in space or 
linked to components (e.g., modeled building objects). In this way it is configured as a real DT platform. 


As for the spatial engine, this receives the necessary information and thus the positions of workers and moving 
objects (provided by UWB or GPS-RTK). The purpose of the spatial engine is to perform real-time processing 
about spatial interference and then transmit it to the platform, which will also relate it to the components of 
buildings in space so as to identify objects of obstruction to operator's view. 
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Finally, the last component of the system is an MR head-mounted display whose use has two purposes. The first 
is to enable the display of holograms for a diminished reality visualization. The second is to provide the position 
and orientation of the observer. The Microsoft Hololens used as the MR provider receives the information directly 
from the platform and thus the visualization of the skeleton joints, the model of the surrounding environment, the 
positions of moving objects, and objects identified as obstacles and whose representation is then modified by the 
procedure to change from solid to wireframe. In order to achieve a consistent visualization, the Hololens transmits 
to the platform the position of the operator with respect to the modelled space and information about the orientation 
of the head. In this way it is possible to precisely calibrate the sample scenario. 


6. DIMINISHED REALITY APPLICATION 

The experiment was conducted using a Python application sensing the skeleton information coming from the depth 
camera ZED 2i through the Python SDK. In this application, the camera acquired two video streams at 30 fps, and 
once every 3 frames the skeletons information where encoded as JSON objects and published using the MQTT 
protocol. For debug purposes, the position of workers' skeleton joints where highlighted in a live video stream 
(Fig. 6 b). This resulted in 10 MQTT messages sent per second. The DT Platform subscribed the topic on the 
MQTT broker, thus receiving the skeleton signal from the Python application. The DT platform was in charge of 
sending the skeleton signal as well as the BIM design and the location of the camera, the Head Mounted Display, 
and the object moved by the crane, to the Spatial Engine. The latter was responsible for detecting in real-time the 
list of obstacles in the BIM design and passed it to the Head Mounted Display, together with the position of the 
workers and the crane load w.r.t. the Head Mounted Display. 


The logic behind the Spatial Engine detects obstacles using ray-casting, a technique implemented by the Spatial 
Engine which returns a list of objects hit by a hypothetical ray departing from a source position towards a given 
direction, and having a length bounded by a given maximum value. To reach our aim, we need to use the user's 
viewpoint as source position of the ray-to-be-casted, next we need to cast a ray on each direction corresponding to 
each joint of the workers skeletons detected by the depth camera and each object moved by the crane; finally, we 
must set as maximum length of the casted rays the actual distance between the user and the joints or moved objects. 


7. DEVELOPMENT AND CASE STUDY 


Feasibility tests of the developed application were carried out in the DC3 laboratory of the DICEA department of 
the Polytechnic University of Marche. In this case since a crane was not available, the overhead crane carriage 
inside the laboratory was used to simulate the suspended load in motion. The hook of the bridge crane was 
instrumented with a UWB position sensor which sends data to the platform (Figure 4). The laboratory is equipped 
with UWB anchors placed at the corners of the room for precise position sensing. 
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Fig. 4: a) Bridge crane hook instrumented with UWB sensor; b) UWB sensor visualization in the space inside 
the platform. 


Next, the ZED 2i camera was placed as seen in Figures 5 and 6 a). The information that the chamber returns 
through the recognition of the skeleton joints is as in Figure 6 b). The orientation of the worker is also detected as 
the blue joints detect the left side of the body while the red joints detect the right side. In this way it is possible to 
understand which direction the worker is looking at and thus whether in the presence of a moving load he is 
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presumably aware of the presence of the load or whether his being from behind identifies a situation of greater 
danger as he may not have noticed that something is moving in his vicinity. 


SUSPENDED LOAD 


Fig. 5: Components position inside the laboratory. 


The test performed reproduced the same conditions shown in Figure 1. The laboratory is placed in a shed divided 
in half by a wall 3.5 m high, about half of the total height of the structure. Figure 5 shows the location of all 
components for the experiment in the laboratory. In order to verify the actual operation of the application for the 
developed diminished reality, the operator wearing the HoloLens stood on the left side in the floor layout of the 
shed. Looking toward the partition wall (Fig. 7 a)) the HoloLens showed the wall in wireframe visualization and 
the skeleton points of the operator who was moving around the load on the other side of the room. In this first 
processing the positioning of the load was not implemented in the hologram visualization although it was present 
as data in the platform. Again, it is possible to observe the differentiation of blue and red colours for the left and 
right side of the body respectively which allows the worker observing the scene to understand the orientation of 
the worker. 


Fig. 6: a) Load position and ZED 2i location; b) Depth camera vision and skeleton joints recognition. 
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Fig. 7: a) Vision from inside the HoloLens (other side of the wall) ; b) Vision from inside the HoloLens (same 
side of the worker framed); c) operator with the HoloLens and the bridge-crane remote control (other side of the 
wall). 


8. CONCLUSION 


This paper presents an application of diminished reality for safety management support during operations at a 
construction site. The proposed system involves the use of depth camera and sensors for locating components of 
interest and implements the development of a DT platform for real-time management. The system thus developed 
allows for optimization in the number of sensors that can be placed in strategic areas (depth camera) and alternately 
on moving loads or equipment to be monitored at the stages when they are expected to be utilized. Initial 
experiments of the application of this development were carried out in the laboratory by reproducing conditions 
similar to those at a real construction site. The next steps will concern the implementation of the visualization 
inside the HoloLens of the position of the moving load and tests in outdoor construction sites thus having the 
possibility to test also the visual rendering of holograms in the open air that could be an obstacle to the optimal 
visualization for operators. 
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